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Abstract Body 


Problem / Background / Context: 

Concerns persist that not enough students are aequiring the complex knowledge and skills that 
transfer to the novel situations they will eonfront in eollege, eareer and eivie life. As a result, 
there has been inereased interest in researeh to evaluate whether a deeper learning approaeh is 
assoeiated with improved edueational outcomes for students of all baekgrounds. However, to 
date there is little rigorous empirieal evidence on whether deeper learning approaehes produce 
better student outeomes (National Researeh Couneil (NRC), 2012; Yuan & Le, 2010). 

Purpose / Objective / Research Question / Focus of Research: 

The Study of Deeper Learning: Opportunities and Outcomes, funded by the William and Flora 
Hewlett Foundation, is a proof-of-concept study to determine whether students attending high 
sehools with a mature and at least moderately well-implemented approaeh to promoting deeper 
learning experienee greater deeper learning opportunities and outcomes than they would have 
had they not attended these sehools. The study ineludes a set of high sehools belonging to 
networks that are aligned with a deeper learning approaeh. In paper #2 of this proposed symposium, 
we have established that students in network high sehools did indeed experienee greater 
opportunities to learn deeply, that those opportunities were broadly spread aeross students, and 
that deeper learning opportunities were assoeiated with deeper learning outeomes. 

The purpose of this paper is to provide quasi-experimental empirieal evidenee on whether or not 
students who experienced these network schools’ approaches aehieved better outcomes than they 
would have otherwise. Specifieally, did students attending these network schools 

1) score higher on measures of eontent knowledge and eritieal thinking skills assoeiated 
with the OECD PISA-based Test for Sehools 

2) report higher levels of intra- and inter-personal eompeteneies and dispositions 

3) attain a higher rate of graduation or 

4) attain a higher rate of post-secondary matriculation 

than they would have if they had not attended the network sehools? 

Improvement Initiative / Intervention / Program / Practice: 

As discussed in the abstract for paper #2, the Study of Deeper Learning examined a sample of 
network schools that were implementing instructional approaches aligned with deeper learning. 
The networks and schools differed in their approaches to deeper learning, but some features were 
common across network schools. For example, all schools engaged students in some form of 
project-based learning that focused on the development of problem-solving skills, collaboration 
skills, and/or communication skills, while engaging students in real-world experiences. Schools 
were engaged in authentic assessment practices including portfolios and exhibitions, and teachers 
used frequent formative assessments to gauge student learning and inform instruction. In 
addition, schools provided personalized learning environments to engage students in learning and 
provide individualized support. 

Setting: 

As in the previous papers, this analysis relies on student-level data from schools located in six 
districts across two states: California and New York. 
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Population / Participants / Subjects: 

As in the previous papers, the sample of sehools for this analysis ineluded a set of moderate to 
high-implementing network high sehools and a set of sehools serving similar student populations 
in the same jurisdiction as the network school, but not belonging to one of the 10 networks. Our 
goal was to identify and recruit a “non-network” school with similar student characteristics for 
each network school so that we could conduct student-level analyses with similar student 
populations across the two groups of schools. Specific criteria for sampling of the network and 
non-network schools are included in the abstract for paper #1. 

For the analysis and findings described in this paper, we collected data from students within 12 
network high schools, which represented 8 of the 10 networks and were located in five different 
districts across two states.^ Analyses also include students attending 10 non-network schools 
located in six districts across two states. One non-network school was matched to two network 
schools, for a total of 1 1 school pairs. 

Research Design: 

See the abstract for paper #1 in this symposium for a detailed summary of the research design, ft 
is important to note again that this quasi-experiment has several limitations. As a proof of 
concept study, designed for internal validity within a focused set of schools, this study is not 
designed to be generalizable across all possible contexts. Due to the type of design, the study is 
vulnerable to unmeasured selection bias. We matched and weighted students on a series of 
measured characteristics (8* grade incoming achievement, etc.), but the design does not account 
for any unmeasured characteristics associated with students’ attendance at network schools. 
Further, as discussed below, although the measures have reasonable reliability and quality, at the 
time of data collection, the field lacked extensively validated measures of some of the deeper 
learning constructs we sought to measure. 

Data Collection and Analysis: 

This study presented several measurement and data collection challenges. We aimed to examine 
a range of deeper learning competencies: mastery of core academic content, critical thinking and 
complex problem-solving, effective communication skills, collaboration skills, ability to learn how to 
leam, and development of academic mindsets and dispositions. However, the field lacks established 
assessments for all of these competencies. For instance, existing state tests do not typically 
measure the deeper learning competencies, and there are few validated measures of the non- 
cognitive outcomes hypothesized to support college and career success. Furthermore, because 
deeper learning is expected to result in improved graduation rates, college attendance, and 
ultimately career success, the study required us to track student cohorts to the end of high school 
and beyond to examine postsecondary outcomes. 

In response to these challenges we conducted multiple data collections: 


^ Two of these network schools were combined for analyses since they had small student populations eligible for 
the study and were co-located on one campus. 

^ An additional eight network schools participated in the study but were excluded from this analysis either because 
1) we did not identify a matched comparison school that met our criteria and was willing to participate, or 2) we 
were unable to obtain parent consent for a sufficient number of students for certain data collections. 
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• OECD PISA-based Test for Schools: To assess students’ content knowledge, and 
critical thinking and problem solving skills, we administered the OECD PISA-based Test 
for Schools to all sampled and consented 11* and 12* grade students in each network and 
non-network school in spring 2013. For the purposes of this research we received 
approval to use the test with older age students than the international norming sample and 
to use Maximum Likelihood Estimates for individual students as our primary outcome. In 
total we administered the assessment to 1,267 students, with an average response rate of 
61%.^ 

• Student survey data: To measure the intra- and inter-personal competencies and 
dispositions that are considered important to college and career readiness, we 
administered a one-hour survey to all sampled and consented 11* and 12* grade students 
in each network and non-network school in spring 2013. For a list of the constructs 
measured in the survey, see Appendix Exhibit 1 . The survey enabled us to gather data 
from a large representative sample of students across both network and non-network 
schools. In total, we administered surveys to 1,762 students from the 1 1 pairs of schools 
with an average response rate of 76%. 

• Extant District Data: We collected district administrative records to examine effects of 
network school attendance on high school graduation within four years (for students who 
graduated within the same district) and achievement test scores. 

• National Student Clearinghouse data: We requested data from the National Student 
Clearinghouse (NSC) for all students who began the 9* grade within our selected schools 
between 2007-08 and 2009-10. NSC data allow us to measure postsecondary enrollment, 
institution type (i.e., two-year, four-year, selective institution"^), and persistence into the 
second year of college. 

The quasi-experimental study also presented several analytic challenges that are described in the 
first paper of this symposium. Briefly, in response to those challenges, we analyzed data 
collected from students within pairs of similar network and non-network schools operating in the 
same or similar district contexts. We used propensity score matching to identify students in the 
comparison schools who at entry to 9* grade matched those in the treatment schools. We 
followed 5 cohorts of students - those entering 9* grade in fall of 2007 to 201 1 . For the results 
reported in this presentation, we focused on a survey and assessment administered to students in 
11* and 12* grade in 2012-13. We performed doubly robust ordinary least squares regression 
models and logistic models for binary outcomes that applied propensity score weights and 
accounted for student demographics and pre-high school (i.e., middle school) achievement. In 
addition to propensity score weights, we also calculated and applied weights that accounted for 
consent, sampling, and non-response. We performed student-level analyses separately within 
each pair of network and non-network schools, and then calculated the precision-weighted 
average difference between network and non-network students across the 1 1 pairs of schools 
using a fixed effects meta- analysis. 


^ One pair of schools was included in the survey sample but not the OECD PISA-based Test for Schools sample. 

Information about institutional selectivity was merged using data from the Integrated Postsecondary Education 
Data System. Selective institutions are defined as four-year institutions in which at least 80 percent of students are 
full-time students, and students' test scores place the institution within the top quintile of baccalaureate-granting 
institutions in the United States. 
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Findings / Outcomes: 

Below, we summarize the overall results of our analyses of student survey and OECD PISA- 
based Test for Sehools Data, based on a meta-analysis that pools results aeross the 1 1 school 
pairs. The results are shown in Appendix Exhibit 2. The primary findings include: 

• Students in participating network high schools scored significantly higher on the OECD 
PISA-based Test for Schools in reading, mathematics , and science than students in the 
paired non-network schools. Effect sizes for all three test subjects were similar in 
magnitude, with the scores of students attending network schools exceeding the scores of 
students in non-network schools by approximately 0.1 1 standard deviations. 

• Network students reported higher levels of academic engagement, motivation to learn, 
self-efficacy and collaboration skills relative to similar students who attended 
comparison schools. Effect sizes ranged from 0.12 standard deviations (for motivation to 
learn) to 0.20 standard deviations (for academic engagement). On the other intrapersonal 
competencies (perseverance, self-management, creative thinking skills, and locus of 
control) there were no significant differences between network and comparison students. 

In addition to the results presented above, we are currently analyzing district extant data so that 
we can measure the effect of attending a network school on state achievement test scores and on 
attainment of graduation. We have also obtained data from the National Student Clearinghouse 
(NSC) for the oldest cohorts of students so that we can observe whether attending a network 
school influenced postsecondary enrollment, postsecondary persistence, and the selectivity of the 
institutions that students attended. These preliminary findings are currently embargoed until 
internal reviews are completed but will be available in the summer allowing for presentation at 
the conference in the fall. 

Conclusions: 

Rigorous research on deeper learning has lagged behind the political and educational interest in 
this instructional approach. As deeper learning gains momentum across the country as a means to 
prepare all students for college and careers, it is essential that policymakers and practitioners 
have evidence of its effectiveness. This quasi-experimental study starts to build this empirical 
evidence base by indicating positive effects of students’ attendance in a deeper learning network 
school on the content knowledge and complex problem solving skills measured by the OECD 
PISA-based Test for Schools, as well as on several intra- and interpersonal competencies and 
dispositions that are considered important for college and career readiness. This study provides 
promising initial evidence that a deeper learning approach may have the potential to make a 
difference in students’ educational experiences and academic skills. The findings in this study 
suggest the potential merit of further research to replicate the findings with a larger sample of 
schools across a wider range of contexts. Additional research recommended includes following 
students through college and beyond to determine longer-term effects, and conducting an 
experimental study with a lottery assignment of students to schools. 
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Appendix B. Tables and Figures 


Exhibit 1 ; Rasch Reliabilities for eonstruets measured in the student survey 

Rasch 

reliability 

Opportunities for deeper learning 

Opportunity for Complex Problem Solving 0.9 

Opportunity to Communicate (combined) 0.83 

Opportunities to Collaborate 0.69 

Opportunity to Learn How to Learn 0.52 

Assessments Aligned with Deeper Learning 0.77 

Feedback to Students 0.75 

Interdisciplinary learning 0.78 

Opportunities for Creative Thinking 0.79 

Real World Connections 0.84 

Dispositional outcomes 

Collaborative Orientation 0.83 

Creative Orientation 0.77 

Perseverance 0.79 

Self-Management 0.81 

Academic Engagement 0.74 

Motivation to Learn 0.75 

Self-Efficacy 0.84 

Locus of Control 0.73 
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Effect Size 


Exhibit 2: Estimated Network School Effects for Outcomes 

Estimated Network School Effects: Outcome Meta-Analysis Results 
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